Search for: All records

We propose a novel framework that leverages Visual Question Answering (VQA) models to automate the evaluation of LLM-generated data visualizations. Traditional evaluation methods often rely on human judgment, which is costly and unscalable, or focus solely on data accuracy, neglecting the effectiveness of visual communication. By employing VQA models, we assess data representation quality and the general communicative clarity of charts. Experiments were conducted using two leading VQA benchmark datasets, ChartQA and PlotQA, with visualizations generated by OpenAI’s GPT-3.5 Turbo and Meta’s Llama 3.1 70B-Instruct models. Our results indicate that LLM-generated charts do not match the accuracy of the original non-LLM-generated charts based on VQA performance measures. Moreover, while our results demonstrate that few-shot prompting significantly boosts the accuracy of chart generation, considerable progress remains to be made before LLMs can fully match the precision of human-generated graphs. This underscores the importance of our work, which expedites the research process by enabling rapid iteration without the need for human annotation, thus accelerating advancements in this field. 

We propose a novel framework that leverages Visual Question Answering (VQA) models to automate the evaluation of LLM-generated data visualizations. Traditional evaluation methods often rely on human judgment, which is costly and unscalable, or focus solely on data accuracy, neglecting the effectiveness of visual communication. By employing VQA models, we assess data representation quality and the general communicative clarity of charts. Experiments were conducted using two leading VQA benchmark datasets, ChartQA and PlotQA, with visualizations generated by OpenAI’s GPT-3.5 Turbo and Meta’s Llama 3.1 70B-Instruct models. Our results indicate that LLM-generated charts do not match the accuracy of the original non-LLM-generated charts based on VQA performance measures. Moreover, while our results demonstrate that few-shot prompting significantly boosts the accuracy of chart generation, considerable progress remains to be made before LLMs can fully match the precision of human-generated graphs. This underscores the importance of our work, which expedites the research process by enabling rapid iteration without the need for human annotation, thus accelerating advancements in this field. 

Abstract Joint inversion of multiple data types was studied as early as 1975 in [K. Vozoff and D. L. Jupp,Joint inversion of geophysical data,Geophys. J. Internat. 42 1975, 3, 977–991],where the authors used the singular value decomposition to determine the degree of ill-conditioning of joint inverse problems. The authors demonstrated in several examples that combining two physical models in a joint inversion, and by effectively stacking discrete linear models, improved the conditioning as compared to individual inversions. This work extends the notion of using the singular value decomposition to determine the conditioning of discrete joint inversion to using the singular value expansion to determine the well-posedness of joint operators. We provide a convergent technique for approximating the singular values of continuous joint operators. In the case of self-adjoint operators, we give an algebraic expression for the joint singular values in terms of the singular values of the individual operators. This expression allows us to show that while rare, there are situations where ill-posedness may be not improved through joint inversion and in fact can degrade the conditioning of an individual inversion. The expression also quantifies the benefits of including repeated measurements in an inversion. We give an example of joint inversion with two moderately ill-posed Green’s function solutions, and quantify the improvement over individual inversions. This work provides a framework in which to identify data types that are advantageous to combine in a joint inversion.